[SC-13271] Pass test parameters to LLM test result description generation #451
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Description
What and why?
What
Pass test parameters through the LLM description generation pipeline so it can be included in the user prompt.
Why
Pass test parameters to the LLM description generation so that descriptions can reference parameter values (e.g., thresholds) that affect test results but aren’t included in the output tables. For example, tests like
TooManyZeroValuesuse thresholds to determine pass/fail, but without access to the actual parameter value, the LLM assumes the default threshold. This can create inconsistencies between the pass/fail outcome and the results, and may lead to hallucinations due to misinterpreting the underlying data.How to test
Run tests with parameters and check that parameters are part of the user message and that the generated description references them. This can be checked in Langfuse traces:
What needs special review?
Dependencies, breaking changes, and deployment notes
https://github.com/validmind/backend/pull/2255
Release notes
Checklist